Laser enhanced reconstruction of 3d surface

ABSTRACT

A method for reconstructing a surface of a three-dimensional object (41) involves a projection of a laser spot pattern (12, 14) onto the surface of the three-dimensional object (41) by a laser (11), and a generation of a series of endoscopic images (24) as an endoscope (21) is translated and/or rotated relative to the three-dimensional object (41). Each endoscopic image (24) illustrates a different view (23) of a laser spot array (13, 15) within the laser spot pattern (12, 14) as projected onto the surface of the three-dimensional object (41) by the laser (11). The laser spot array (13, 15) may be identical to or a subset of the laser spot pattern (12, 14). The method further involves a reconstruction of the surface of the three-dimensional object (41) from a correspondence of the different views (23) of the laser spot array (13, 15) as illustrated in the endoscopic images (24).

CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is a Continuation of application Ser. No. 13/577,456,dated Aug. 7, 2012, which is the U.S. National Phase application under35 U.S.C. § 371 of International Application No. PCT/IB2011/050171,filed on Jan. 14, 2011, which claims the benefit of U.S. ProvisionalPatent Application No. 61/303,702, filed Feb. 12, 2010. Theseapplications are hereby incorporated by reference herein.

The present invention generally relates to a reconstruction of athree-dimensional (“3D”) surface of an object during a minimallyinvasive endoscopic surgery. The present invention specifically relatesto a generation, detection, and utilization of reproducible and precisefeatures of a laser spot pattern on a surface of an object forintra-operative camera calibration of an endoscope and for 3Dreconstruction of the surface of the object.

Minimally invasive endoscopic surgery is a surgical procedure in which arigid or flexible endoscope is introduced into a patient's body througha natural orifice or a small incision in the skin (i.e., a port).Additional surgical tools are introduced into the patient's body throughsimilar ports with the endoscope being used to provide a visual feedbackto a surgeon of the surgical tools as related to the surgical site.Examples of minimally invasive endoscopic surgery include, but are notlimited to, an endoscopic heart surgery (e.g., an endoscopic cardiacbypass or a mitral valve replacement), a laparascopy for the abdomen,arthroscopy for joints, and bronchoscopy for the lungs.

Laser metrology in endoscopy is a class of methods providing apossibility to measure size of objects in the endoscopy images. It isusually done for industrial endoscopy, but it is also used in medicine.In laser metrology, a collimated laser beam is positioned parallel tothe optical axis of the endoscopic camera. The laser beam projects alaser dot on the object or near the object under consideration. Due tothe beam collimation of the laser, the size of the dot on or nearby theobject remains the same independent of a distance to the object. Thus,the dot size serves as a size calibrator for the image.

One known method in the art places a laser-generating device on thedistal end of the endoscope to generate a calibrated laser dot for lasermetrology. Basically, this laser metrology method projects one (1)collimated laser dot on the object (e.g., tissue) and retrieves a scaleof the object from a diameter of the laser dot.

By comparison, another known method in the art utilizes an endoscopewith four laser beams set parallel to the optical axis of an endoscopiccamera to project four (4) laser dots on the object (e.g., tissue) tofind scale of the object in the image. For this method, radialdistortion compensation is performed using a chessboard-like calibrationgrid to obtain distortion parameters. Subsequently, 3D position of laserdots may be computed from geometrical relations between the points usinglens geometry. Finally, calibration rulers are displayed on theendoscopic images.

As previously stated herein for minimally invasive endoscopic surgeries,endoscopes are providing the only visual feedback of the operating site.However, endoscopic images are usually two-dimensional (“2D”), whichposes difficulties in obtaining depth information as well as a relativeposition and size of the objects in the view. Known algorithms forreconstruction of 3D surfaces from a series of 2D images rely on findingcorrespondence between points in two or more frames. The quality of 3Dreconstruction from such algorithms depends heavily on the accuracy ofthe matched features. In particular, in order to reconstruct 3D surfacefrom 2D+t series, using RANdom SAmple Consensus (“RANSAC”) optimizationeight (8) or more feature-matches have to be found. However, in surgery,objects in the endoscope view are very often smooth and featureless(e.g., cardiac tissue in cardiac endoscopy or bone surface inarthroscopy), which makes feature detection and matching a difficulttask.

The aforementioned laser metrology methods solve the scale problem(i.e., object size) by using a single collimated laser dot or multiplelaser beams positioned parallel to the optical axis of the scope.However, these methods do not address the quality issue of 3Dreconstruction. Another downside of these methods is that they requirethe laser beam to be parallel to the optical axis of the endoscope.Thus, the laser source and endoscopic fibers have to be integrated intoendoscope itself, which increases the diameter of the endoscope, thisincreasing invasiveness of the surgical procedure.

The present invention utilizes a laser for projecting a laser spotpattern (e.g., a matrix of circular dots on a surface of a 3D object(e.g., an organ or a tissue of interest) to facilitate a precisereconstruction of the surface of the object and an intra-operativecamera calibration that overcomes the difficulties from a 2D endoscopicview in obtaining depth information as well as relative position andsize of the surface of the object.

One form of the present invention is a system employing a laser, anendoscope and an image reconstruction device. In operation, the laserprojects a laser spot pattern (e.g., a matrix of circular dots) onto asurface of a 3D object (e.g., an organ or tissue of interest). Theendoscope generates a series of endoscopic images as the endoscope istranslated and/or rotated relative to the 3D object with each endoscopicimage illustrating a different view of a laser spot array within thelaser spot pattern as projected onto the surface of the 3D object by thelaser. The image reconstruction device reconstructs the surface of the3D object from a correspondence of the differing views of the laser spotarray as illustrated in the endoscopic images.

For purposes of the present invention, the term “laser spot pattern” isbroadly defined herein as any spatial arrangement of two or more laserspots of any geometrical form, of any color and of any practicaldimensions for an endoscopic application, and the term “laser spotarray” is broadly define herein as having the spatial arrangement oflaser spots of an associated laser spot pattern or any subset thereof.Within a laser spot pattern and a laser spot array, the geometricalform, color and dimension of each spot may be identical or vary amongsome or all of the laser spots. Additionally, the laser spot array maybe pre-operatively or intra-operatively defined within the laser spotpattern.

Furthermore, the term “endoscope” is broadly defined herein as anydevice having the ability to image from inside a body. Examples of anendoscope for purposes of the present invention include, but are notlimited to, any type of scope, flexible or rigid (e.g., endoscope,arthroscope, bronchoscope, choledochoscope, colonoscope, cystoscope,duodenoscope, gastroscope, hysteroscope, laparoscope, laryngoscope,neuroscope, otoscope, push enteroscope, rhinolaryngoscope,sigmoidoscope, sinuscope, thorascope, etc.) and any device similar to ascope that is equipped with an image system (e.g., a nested cannula withimaging). The imaging is local, and surface images may be obtainedoptically with fiber optics, lenses, or miniaturized (e.g. CCD based)imaging systems.

The foregoing form and other forms of the present invention as well asvarious features and advantages of the present invention will becomefurther apparent from the following detailed description of variousembodiments of the present invention read in conjunction with theaccompanying drawings. The detailed description and drawings are merelyillustrative of the present invention rather than limiting, the scope ofthe present invention being defined by the appended claims andequivalents thereof.

FIG. 1 illustrates an exemplary embodiment of a 3D image reconstructionsystem in accordance with the present invention.

FIG. 2 illustrates one exemplary series of different views of a laserspot array in accordance with the present invention.

FIG. 3 illustrates another exemplary series of different views of thelaser spot array shown in FIG. 2 in accordance with the presentinvention.

FIG. 4 illustrates a flowchart representative of an exemplary embodimentof an endoscopic surgical method in accordance with the presentinvention.

FIG. 5 illustrates an exemplary endoscopic application of the methodshown in FIG. 4 by the system shown in FIG. 1.

FIG. 6 illustrates a flowchart representative of an exemplary embodimentof a 3D surface reconstruction method in accordance with the presentinvention.

An implementation of 3D surface reconstruction algorithms by the presentinvention is accomplished by a laser projecting a laser spot pattern ona 3D object and an endoscope generating a series of 2D endoscopic imagesof a laser spot array within the laser spot pattern. The laser spotpattern serves as a reproducible and precise feature as projected on the3D object to facilitate a correspondence of the laser spot array amongthe endoscopic images.

For example, as shown in FIG. 1, a laser 11 powered by a laser source 10projects a laser spot pattern 12 having a 7×7 matrix arrangement ofcircular dots. Within laser spot pattern 12 is a laser spot array 13having a 3×3 matrix arrangement of circular dots. During a minimallyinvasive endoscopic surgery, an endoscope 21 is focused on an entiretyor a portion of laser spot pattern 12 whereby a field of view 22 ofendoscope 21 encircles laser spot array 13.

More particularly, FIG. 2 illustrates a sequence of endoscopic views 23a-23 c generated by endoscope 21 as endoscope 21 is translated in adirection of the 3D object. As such, laser spot pattern 12 enlarges inthe different endoscopic views 23 a-23 c with laser spot array 13 beingidentifiable in each endoscopic view 23 a-23 c. The enlargement of laserspot pattern across endoscopic view 23 a-23 c serves as a motion oflaser spot array 13 for purposes of implementing the 3D surfacereconstruction algorithms as will be further explained herein.

FIG. 3 illustrates an additional sequencing of endoscopic views 23 d-23f generated by endoscope 21 as endoscope 21 is further translated in adirection of the 3D object. Again, laser spot pattern 12 enlarges acrossthe endoscopic views 23 d-23 f with laser spot array 13 beingidentifiable in each endoscopic view 23 d-23 f, and the enlargement oflaser spot pattern across endoscopic views 23 d-23 f serves as a motionof laser spot array 13 for purposes of implementing the 3D surfacereconstruction algorithms as will be further explained herein. FIGS. 2and 3 are provided to emphasize the inventive principle of a laser spotpattern as a reproducible and precise feature as projected on a 3Dobject to facilitate a correspondence of a laser spot array among theendoscopic images. In practice, preferably the laser spot patternincludes nine (9) or more laser spots, the number of endoscopic imagesis two (2) or more, and the distance between the endoscopic camera andthe laser spot pattern facilitates an identification of the entire laserspot pattern across all of the endoscopic images whereby the laser spotpattern itself serves as the laser spot array. Nonetheless, FIGS. 2 and3 highlight that a laser spot array, whether an entirety or a portion ofthe laser spot pattern, must be identifiable among all of the endoscopicimages.

Referring back to FIG. 1, an image reconstruction device 20 processesthe generated images of the laser spot array to reconstruct a 3D imageof the surface of the object. For purposes of the present invention, animage reconstruction device is broadly defined herein as any devicestructurally configured for generating a 3D reconstruction of a surfaceof an object by processing endoscopic images in accordance with 3Dreconstruction algorithms (e.g., a programmed computer), and the term“generating” as used herein is broadly defined to encompass anytechnique presently or subsequently known in the art for creating,computing, supplying, furnishing, obtaining, producing, forming,developing, evolving, modifying, transforming, altering or otherwisemaking available information (e.g., data, text, images, voice and video)for computer processing and memory storage/retrieval purposes,particularly image datasets and video frames.

FIG. 4 illustrates a flowchart 30 representative of an endoscopicsurgical method of the present invention. Flowchart 30 includespre-operative stages S32 and S33, and intra-operative stages S34-S37.The term “pre-operative” as used herein is broadly defined to describeany activity occurring or related to a period or preparations before anendoscopic application and the term “intra-operative” as used herein isbroadly defined to describe as any activity occurring, carried out, orencountered in the course of an endoscopic application (e.g., operatingthe endoscope). Examples of an endoscopic application include, but arenot limited to, an arthroscopy, a bronchoscopy, a colonscopy, alaparoscopy, a brain endoscopy, and an endoscopic cardiac surgery.Examples of an endoscopic cardiac surgery include, but are not limitedto, endoscopic coronary artery bypass, endoscopic mitral and aorticvalve repair and replacement.

Pre-operative stage S31 encompasses a selection of a laser forprojecting the laser spot pattern on the 3D object. In practice, aLasiris™ SNF laser may be used for endoscopic applications whereby thelaser has a wavelength approximately 600 nm and a power less than 100mW. Further, the laser preferably projects laser spot pattern a green orblue 7×7 matrix of circular dots whereby eight (8) or more of thecircular dots may serve as the laser spot array. Further, the circulardots may have a 0.5 mm diameter with a 4 mm spacing between the circulardots. To specify a fan angle (FA) of ninety (90) degrees or less, anobject size (L) and an operating distance (D) must be know in accordancewith the following equation [1]:

FA=2*arcsin(L/(2*D))  [1]

FIG. 5 illustrates an example of an arthroscopic application involvinglaser 11 being a distance D from tissue 41 of a knee 40 with an objectsize L.

Referring again to FIG. 4, pre-operative stage S32 encompasses a knowncamera calibration of the endoscope. In one embodiment, the laserprojects the laser spot pattern on a contrasting planar surface (e.g., awhite planar surface) with the laser spot pattern having a uniformmatrix (e.g., 7×7 matrix of circular dots) or a matrix with differentnumber of laser spots in the two dimensions (e.g., 6×7 matrix ofcircular dots). The calibration parameters as well as radial distortionare estimated by the acquiring of images of the laser spot pattern onthe planar surface under two (2) or more different orientations of theendoscope relative to the planar surface, and by a detection of thelaser points in the images.

Intra-operative stage S33 encompasses a generation of the laser spotpattern on the surface of the 3D object. For example, as shown in FIG.5, laser 11 is inserted within a surgical instrument port 43 of knee 40to thereby project a laser spot pattern 14 of a 5×5 matrix of circulardots onto tissue 41.

An execution of intra-operative stage S34 is dependent of whether stageS32 was not executed during the pre-operative phase, or if are-calibration of the endoscope is required. If executed,intra-operative stage S32 encompasses an endoscope taking images of thelaser spot pattern projected onto the 3D object under two (2) or moredifferent orientations of the endoscope relative to the laser spotpattern. For example, as shown in FIG. 5, an endoscope 21 is inserted ina visual port 42 of knee 40 to thereby generate an image of laser spotpattern 14 (e.g., an image 24). The endoscope 24 can be moved aroundport 42 (i.e., a pivot point) at any direction and rotation to generateimages of the same laser spot array from different viewing angles anddirections. Thereafter, a detection of the laser spot pattern 14 as thelaser spot array within the images or a detection of a laser spot array15 within the images would enable an estimation of the cameraparameters.

Detection of laser spots can be performed with any algorithm known inart, such as color thresholding. Result of the detection is x[x,y]^(T)position of the spot in a coordinate system of each image.

Intra-operative stage S35 encompasses a generation of a series of two(2) or more images of the laser spot pattern on the 3D object as theendoscope is translated and/or rotated relative to the 3D object and theport 42. For example, as shown in FIG. 5, endoscope 21 is inserted inport 42 of knee 40 to thereby generate images of laser spot pattern 14(e.g., an image 24) as endoscope 21 is translated in a direction oftissue 41 as shown by the arrow.

Intra-operative stage S36 encompasses a 3D reconstruction of the surfaceof the object from the endoscopic images acquired during stage S35 andthe calibration of the endoscope obtained during pre-operative stage S32or intra-operative stage S34. In practice, any 3D reconstructionalgorithm may be implemented during stage S36 to achieve the 3Dreconstruction of the object. In one embodiment, a shown in FIG. 6, aflowchart 50 representative of a 3D surface reconstruction that may beimplemented during stage S36.

Referring to FIG. 6, a stage S51 of flowchart 50 encompasses ageneration of a fundamental matrix (F) for relating the different viewsof the laser spot array across the endoscopic mages. In one embodiment,for the same laser spot in two different views (x) and (x′), fundamentalmatrix (F) is a 3×3 matrix and is defined in accordance with thefollowing known equation [2]:

X ^(T) *F*x′=0  [2]

For N laser spots in two different views, a set of N equations isdefined:

$\begin{matrix}{{{x_{1}^{T}*F*x_{1}^{\prime}} = 0}\ldots{{x_{N}^{T}*F*x_{N}^{\prime}} = 0}} & \lbrack 3\rbrack\end{matrix}$

The unknown (F) from equations [3] may be computed using an Eight-pointalgorithm if the laser spot array has eight (8) laser spots (N=8), ormay be computed using an iterative method (e.g., RANSAC) if the laserspot array includes nine (9) or more laser spot.

Stage S52 encompasses a generation of an essential matrix (E) orrelating the different views of the laser spot array across theendoscopic mages. In one embodiment, the essential matrix (E) iscomputed from the following known equation [4]:

E=K ^(T) *F*K=0  [4]

Calibration matrix (K) is a 3×3 matrix representative of thepre-operative or intra-operative calibration of the endoscope.

Stage S53 encompasses a generation of a translation vector (T) and arotation matrix (R) (if the endoscope was rotated) as a function of theessential matrix (E). In one embodiment, a translation vector (T) and arotation matrix (R) are derived from the following known equation [5]:

E=U*Σ*V ^(T)=0  [5]

Stage S54 encompasses a generation of a projection matrix for each viewof the laser spot array. In one embodiment for two (2) views of thelaser spot array, a projection matrix P₁ for a view associated withspots (x) and a projection matrix P₁ for a view associated for spots(x′) are computed from the following known equations [6] and [7]:

P ₁ =K*[I|0]  [6]

P ₂ =K ^(T)*[R|T]*K  [7]

Stage S55 encompasses a 3D object point reconstruction from the laserspot array or salient features of the object (e.g., edges) in theendoscopic images. In one embodiment, using a pinhole camera model fortwo (2) views, a 3D object point X is computed from the following knownequations [8] and [9]:

x=P ₁ *X  [8]

x′=P ₂ *X  [9]

The computed 3D object point X may be reconstructed using triangulationand equations [8] and [9].

For points x and x′, two sets could be used for stage S55.

In a first embodiment, laser spots x and x′ can be used as features.These are strong features, because they are highly precise and reliable.This embodiment would result in a very sparse 3D model having as manypoints as the associated laser spot array.

In second embodiment, weak object surface features (e.g., edges)detected using feature detection methods known in art (e.g., a SIFTmethod) may be used with projection matrixes P₁ and P₂ computed frompoints x and x′. This method would result in a dense surface with lowerprecision of points x and x′, but maintaining high precision ofprojection matrixes P₁ and P_(z).

While various embodiments of the present invention have been illustratedand described, it will be understood by those skilled in the art thatthe embodiments of the present invention as described herein areillustrative, and various changes and modifications may be made andequivalents may be substituted for elements thereof without departingfrom the true scope of the present invention. In addition, manymodifications may be made to adapt the teachings of the presentinvention without departing from its central scope. Therefore, it isintended that the present invention not be limited to the particularembodiments disclosed as the best mode contemplated for carrying out thepresent invention, but that the present invention includes allembodiments falling within the scope of the appended claims.

What is claimed is:
 1. A system for reconstructing a surface of athree-dimensional object, the system comprising: an endoscope configuredto generate a series of endoscopic images as the endoscope is translatedand/or rotated relative to the three-dimensional object, wherein eachendoscopic image illustrates a different view of a laser spot arraywithin a laser spot pattern projected onto the surface of thethree-dimensional object; and a processor, in communication with theendoscope, configured to reconstruct the surface of thethree-dimensional object from a correspondence of each different view ofthe laser spot array as illustrated in the endoscopic images.
 2. Thesystem of claim 1, wherein, to reconstruct the surface of thethree-dimensional object, the processor is further configured to:generate a fundamental matrix that relates the different views of thelaser spot array as illustrated in the endoscopic images; andreconstruct three-dimensional object points as a function of thefundamental matrix and the different views of the laser spot array. 3.The system of claim 1, wherein, to reconstruct the surface of thethree-dimensional object, the processor is further configured to:generate a fundamental matrix that relates the different views of thelaser spot array as illustrated in the endoscopic images; detect surfacefeatures of the object as illustrated in the endoscopic image; andreconstruct three-dimensional object points as a function of thefundamental matrix and the surface features of the object detected inthe endoscopic images.
 4. The system of claim 1, wherein, to reconstructthe surface of the three-dimensional object, the processor is furtherconfigured to: generate a fundamental matrix that relates the differentviews of the laser spot array as illustrated in the endoscopic images.5. The system of claim 4, wherein, to reconstruct the surface of thethree-dimensional object, the processor is further configured to:generate an essential matrix that relates the different views of thelaser spot array as illustrated in the endoscopic images, the essentialmatrix being a function of the fundamental matrix and a cameracalibration matrix associated with a camera calibration of theendoscope.
 6. The system of claim 5, wherein, to reconstruct the surfaceof the three-dimensional object, the processor is further configured to:generate a translation vector and a rotation matrix as a function of theessential matrix.
 7. The system of claim 6, wherein, to reconstruct thesurface of the three-dimensional object, the processor is furtherconfigured to: generate a projection matrix for each view of the laserspot array as a function of the translation vector and the rotationmatrix, each projection matrix being a linear transformation of anassociated view of the laser spot array.
 8. The system of claim 7,wherein, to reconstruct the surface of the three-dimensional object, theprocessor is further configured to: reconstruct three-dimensional objectpoints as a function of each projection matrix and associated views ofthe laser spot array.
 9. The system of claim 7, wherein, to reconstructthe surface of the three-dimensional object, the processor is furtherconfigured to: detect surface features of the object as illustrated inthe endoscopic images for each view of the laser spot array; andreconstruct three-dimensional object points as a function of eachprojection matrix and each surface feature of the object detected in theendoscopic images.
 10. The system of claim 1, wherein the endoscope isintra-operatively calibrated from at least two of the endoscopic images.11. The system of claim 1, further comprising: a laser configured toproject the laser spot pattern onto the surface of the three-dimensionalobject.
 12. A method for reconstructing a surface of a three-dimensionalobject, the method comprising: generating a series of endoscopic imagesas an endoscope is translated and/or rotated relative to thethree-dimensional object, wherein each endoscopic image illustrates adifferent view of a laser spot array within a laser spot patternprojected onto the surface of the three-dimensional object; andreconstructing the surface of the three-dimensional object from acorrespondence of each different view of the laser spot array asillustrated in the endoscopic images.
 13. The method of claim 12,further comprising: generating a fundamental matrix that relates thedifferent views of the laser spot array as illustrated in the endoscopicimages; and reconstructing three-dimensional object points as a functionof the fundamental matrix and the different views of the laser spotarray.
 14. The method of claim 12, further comprising: generating afundamental matrix that relates the different views of the laser spotarray as illustrated in the endoscopic images; detecting surfacefeatures of the object as illustrated in the endoscopic image; andreconstructing three-dimensional object points as a function of thefundamental matrix and the surface features of the object detected inthe endoscopic images.
 15. The method of claim 12, further comprising:generating a fundamental matrix that relates the different views of thelaser spot array as illustrated in the endoscopic images; and generatingan essential matrix that relates the different views of the laser spotarray as illustrated in the endoscopic images, the essential matrixbeing a function of the fundamental matrix and a camera calibrationmatrix associated with a camera calibration of the endoscope.
 16. Themethod of claim 15, further comprising: generating a translation vectorand a rotation matrix as a function of the essential matrix.
 17. Themethod of claim 16, further comprising: generating a projection matrixfor each view of the laser spot array as a function of the translationvector and the rotation matrix, each projection matrix being a lineartransformation of an associated view of the laser spot array.
 18. Themethod of claim 17, further comprising: reconstructing three-dimensionalobject points as a function of each projection matrix and associatedviews of the laser spot array.
 19. The method of claim 17, furthercomprising: detecting surface features of the object as illustrated inthe endoscopic images for each view of the laser spot array; andreconstructing three-dimensional object points as a function of eachprojection matrix and each surface feature of the object detected in theendoscopic images.
 20. A non-transitory computer-readable storage mediumhaving stored a computer program comprising instructions, theinstructions, when the computer program is executed by a process, causethe processor to: generate a series of endoscopic images as an endoscopeis translated and/or rotated relative to a three-dimensional object,wherein each endoscopic image illustrates a different view of a laserspot array within a laser spot pattern projected onto a surface of thethree-dimensional object; and reconstruct the surface of thethree-dimensional object from a correspondence of each different view ofthe laser spot array as illustrated in the endoscopic images.